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ABSTRACT • ' 

Data from two studies on evaluation budgeting 
practices are presented. In the first study, estimated budget figures 
are provided for formative and summative evaluations by 29 state 
education agency evaluators. In the second study, initial budget 
figures, final costs, and study characteristics are abstracted from ' 
the records of 85 evaluations conducted under the aegis of a regional 
evaluation .contracting firm. A pattern of budget allocations in the 
two studies was observed in which roughly 61 percent of the total 
evaluation budget was allocated to personnel, 13 to 14 percent to 
overhead, and all other budget categories were comprised of 10 
percent or less of the total budget. These allocation patterns were 
found, to vary , according to certain study characteristics, however. 
Allocation to personnel was related to the size of the total budget , 
the use of a pre-post design, and the number of pag*s in the written 
report. Allocation to the travel budget category was "relate^ to the 
size of the total budget, the distances" traveled, the' conduct of a 
formative evaluation , the provision of evaluation-related services 
such as training, and the use of archival data or a mixed design . 
Data processing allocations were also related to funding level and 
the conduct of a summative evaluation. The. total cost of the 
evaluation was associated with travel requirements , the use of a 
pre-post design , the provision of services , and content . The 
conclusion points to pitfalls in evaluation budgeting practice and 
provides recommendations for future studies . (Author/TE) 
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PREFACE 



The Research on Evaluation Frogr^ua itf & Northwest Regional 
Educational Laboratory project of research, developnent, testing f 
and training designed to create .new evaluation methodologies for 
use in education. This document is one of a aeries of papers and 
reports produced by program staff , visiting scholar 6, adjunct 
scholar s, and project .collaborators — all members of a cooperative 
network of colleagues working on the development of new 
ae thodolog i es * 

. How aura ^ evaluation resources allocated across such standard 
budget categories as personnel, travel, and data processing? Do 
these allocations differ significantly based cm a study's funding 
level or the type of design used? These and related questions 
are answered in this document which reports on two studies of 
evaluation budgeting* The first study looks at projected budget 
allocation patterns for formative and sum&ative evaluations aa 
reported by evaluators in state education agency evaluation 
units. The second study provides a detailed analysis of $5 
actual evaluation budgets of a regional evaluation contracting 
firs. 

Nick L. Smith, Editor 
Paper and Report Series 
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- ABSTBACT 

i Data from two studies on evaluation 'budgeting practices are 
presented. In the first study, estimated budget figures are 
provided for formative and summative evaluations by 29 SEA 
evaiuators. In the second study, initial budget figures, final, 
costs, and. study characteristics are abstracted from the records 
of 85 evaluations conducted under, the aegis of a regional 

m r 

evaluation contracting firm. A pattern of budget allocations in 
the two studies >«3 observed 'in which approximately 61 percent of 
the total evaluation budget was allpcated to personnel, overhead * 
added an additional 13 to 14 percent to the cost of the 
evaluation, and all other budget categories were comprised of 10 
percent or less of the total Jbudget • These allocation patterns 
were found to vary according to certain* study characteristics, 
however. Allocation to personnel was related to ti*e size of the 
total budget, the,. use of a pre-post design, and the number of 
pages 'in the written report. In addition, while e valuators in 
both studies estimated that personnel wquld require more money if 
a formative evaluation were conducted, the actual cost for 
personnel was 9 percent less than projected. Allocation to the 
travel budget category was related to- the si?e of the total 
budget, the distances traveled, the conduct of a formative . 
evaluation, the provision of evalua tea-related services such as 
training, and the use of archival data or a mixed design. Data 
processing allocations were also related to funding level and the 
conduct of a summative evaluation. The total coot of the 
evaluation was associated with travel requirements, the use of a 

pre- post design, the provision of evaluation related services, 

a 

and the content of the evaluation. The conclusion points to 
pitfalls in evaluation budgeting practice and provides 
recommendations for future studies. 



STUDY DESCRIPTION 

IT 

* 

Most evaluatora, trained as researchers rather than managers, 
receive little formal instruction in how to. plan and manage 
evaluation budgets. Rather, skill in handling evaluation budgets 
oust be acquired oh the job where lack of proper monitoring, use 
of "ball park" accounting ay atoms, mistakes of underestimating 
and overestimating costs, and failure to, adjust the evaluation 
work in light of changing financial resources, can be costly and 
result in poor, inadequate evaluation studies. As 'in all areas 
of social service delivery and research, evaluation resources are 
dwindling, making the need to plan properly and manage evaluation 
budgets even more acute. 

Although budgeting is necessary for almost all varieties of 
evaluation practice, there has been little study of it. Alkin 
and Stecher (1963), in reviewing the sparse writing on the 
subject, note that almost no empirical studies have been done on 
the proper allocation of evaluation resources across budget 
categories. Although evaluatoxrs axe urged in the literature to 
conduct evaluations as economically as ^possible, * . nowhere 
is it indicated what dollar amounts or percentages should be 
spent for evaluation, nor are any guidelines ^ffere^ for making 
cost allocations within the evaluation budget" (AikiJKajid^ 
Stecher , 1983, p. 119) . 

i 

This observation of a lack of guidance for evaluation 
budgeting was again 'confirmed in 1935, when our search of the 
Social Science Citation Index and ERIC for 1964 turned up no 
articles on budgeting for evaluation projects. Consequently, we 



8 2 



agree' that the MM; in and Stecher study "... say well represent 
the first coeipr ehansive examination of evaluation coats at the 
•icrolevel" (Alkin and Stecher, 1983, p. 121) . we have therefore 
used their study as t starting point for our work. 

The purpose of tne Alkin~stecher (1983) study was to identify 
typical budget allocation patterns for evaluation and to 
determine the effect of the purpose of the evaluation (formative 
versus summative) and the funding level on. budget allocation 
patterns. Their method was to survey 25 e valuators from * wide 
range of backgrounds and affiliations to obtain .estimates of 
allocations for 7 budget categories for a process and a summative 
evaluation and for funding levels of $4,000, $10,000, and $25,000. 

The Alkin-Stecher study was a useful exploratory 

f 

investigation and provided a good start on the understanding of 
budgeting processes for evaluations. However, there were several 
shortcomings in the study which we believed reduced the 
credibility of their findings. For instance, Alkin and Stecher 
(1933) freely admitted the informal nature of their study, 
labeling thair respondents a . . nonfandom but diverse sample" 
(p. 123 J, and indicated that they*. . . rejected the formalism 
of rigidly structured questionnaires and carefully controlled 
sampling in favor of semi- structured collegia 1 conversations with 
a diverse group of evaluation professionals" (Alkin and Stecher, 
1983, p. 124). 

After looking carefully at the Alkin and Stecher (1983} 
study, we conducted two concurrent studies on budgeting 
procedures for evaluations. These studies were designed as a 
more formal attempt to study arid expand tne- base of understanding 
of budgeting for evaluations* In our first study, we used a mail 
survey to collect budget allocation estimates for formative .^na v 
summative evaluations front a group of ^valuators employed in ^ 
state education agency (SEA) evaluation units. This budget 
information was collected as part of a larger research study (c£« 
Smith and Smith, in press). In our second study, we examined 85 
actual budgets and final reports completed by ©valuators employed 
in a private, nonprofit evaluation contracting firm* 

9 ' 
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Ir. presenting the results of these two studies, we provide a 
profile of how funds are allocated in evaluation budgets and 
discuss the effect of various factors on budget allocation 
patterns and total evaluation costs. We believe that this 
systematic study of evaluation budgeting procedures and the 
identification of pitfalls in the practice of budgeting for 
evaluations will be useful for evaluators learning how to budget 
studies, for clients writing evaluation contracts, and for grant 
review panels judging the feasibility of proposed evaluations. A 
brief overview of the purpose and design of our two studies 
follows. 

STUDIES OF EVALUATION BUDGETING 

Study I; SEA Evaluators 1 Budget Estimates 

The purpose of Study 1 was (1) to obtain a picture of budget 
allocations for evaluation, (2) to determine whether budget 
allocations differed for formative versus summative evaluation 
designs, and (3 J to document method differences in formative 
versus summative evaluations. 

In 1983 a survey was mailed to ail 37 SEAs witn centralized 
program evaluation units. After the initial mailing and three 
follow-up mailings, 29 (78 percent) completed surveys were 
returned. The participating evaluators represented evaluation 
units that conducted an average of 8 evaluations a year and had 
an average 7 full-time professional staff. (See Smith and Smith, 
in press) for a detailed descr iption of the study design and 
sample. } 

The survey provided scenar ios of formative and summative 
evaluations, and asked questions about evaluation activities and 
budgeting for both. The .scenario provided for the formative 
study was as follows: 

Assume that your evaluation unit has been asked to 
evaluate a minority education project in a nearby 
metropolitan city in your state. The project is designed 
to, improve student self-image and school performance, and 

4 
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to increase the ccmaonity acceptance of minority 
students. In this first year of tit* project, your unit 
is to conduct a formative •valuation to provide feedback 
for program improvement to the local school district. 
You have $10,000 total to spend on all expenses of the 
evaluation study* We would like to know what major 
evaluation activities you would include in your study 
(question 1) and how you would allocate the evaluation 
resources (question 2). V 

First, respondents were provided with a list of 18 activities 
generic to any type of evaluation and asked to check each that 
would be a major part of a formative evaluation study with a 
budget of $10,000. Second respondents were asked to allocate the 
$10,000 designated for the formative evaluation to the budget 
categories. Respondents were then provided a second scenario 
explaining that the same program was now in its third year of 
operation and a aummative evaluation has been requested. Again, 
the budget for a summative study was $10,000, the same as for the 
formative study* Information on evaluation activities budget 
allocations was again requested. 

Our budget categories were selected to be comparable to, the 
categories used in both our Study 2 investigation of actual 
projects (described in the next section) and in the Alkin and 
Stecher (1983} study. The budget categories were: 

Personnel costs (professional and support staff salaries 

and benefits) 

Consultant costs (consultant salaries and benefits) 
Staff travel 
Consultant travel 

Communication ( telephone , postage, shipping) 

Facilities (facilities and utilities) 

Duplication (printing and duplication) 

Data processing 

Supplies (materials, supplies, equipment) 

Other services (e.g., subcontracts, conferences please 

specify) v 

In a final question, respondents were asked what indirect or 
overhead rate their agency would normally add to the cost of 
theue evaluations. > 



Study 2i Evaluation project ^Budget Reviews 

The purpose of this study was (I) to describe budget 
allocation patterns in actual evaluation studies, (2) to explore 
the relationship between the cost of specific resources and syuoy 
characteristics^ and (3) to identify pitfalls in evaluation . 
budgeting toy conpar ing initial budget figures with final 
evaluation costs. * 

Budgets froa 85 completed evaluations conducted by evaluators 
employed in a private** nonprofit evaluation contracting firm were 
nade available to us. in many evaluation settings* evaluation 
budgets are not constructed on a project-by-project basis. 
Rather, one agency or unit budget covers the' cost of all 1 projects 

* conducted in a fiscal year; In fact, this is the aost cos»on 
practice for evaltv^tion units in state depar talents and local 

* districts. Consequently', we were fortunate to locate a source of 
available and completed budgets where each evaluation served a 
different client and had its own budget. ► 

All evaluation studies conducted froa 1980 to 1982 (n-85) by 
evaluators in this firm were obtained and abstracted. Host of 
the evaluations wer^ conducted by two evaluators (87%* n*74) . wa 
were given permission to abstract from initially proposed 
budgets* final project expenditure records* and written reports. 
Consequently* for each evaluation* information was obtained on 
(a) the budget as originally estimated, (b) the final budget 
expenditures* and (c) certain characteristics of the study. The 
budget categories used by this agency and for this study were: 

Personnel " (including professional and 

support staff salaries and 
benefits} 

Consultant costs 
* Staff travel 

Consultant travel 

Postage and shipping 

Telephone 

Facilities 

Utilities 

Printing 

Duplication 

Data processing 

12 



Information was obcaii,#a on both the initial budgat and final 
evaluation cost*. A copy "of thai budget abstracting £or» if* 
located in Appendix A. The budget of an average evaluation was 
45,74- (the rang* was $0 to $48,499) , while the final exist of an 
av.rag* evaluation vas $5,602 (the range was $192 to $51,040} . 
In most of the cases, anticipated budget 'coats were comparable to 
actual expenditures* Consequently, in this report we will 
compare the budgeted costs of the evaluation with the estimated 
oudget cost* generated by the SEA e valuators with the respondents 
in the Alkin-Stecher (1383} study. If an initial budgeted coat 
differed notably fro* the contracting firm's final cost, a 
discussion on the reasons for this discrepancy is provided. 

The study characteristics obtained from the evaluation 
written reports included the purpose of the evaluation (e.g?* 
formative versus summative) , the location (e.g., state) and type 
of client {e.g., state or local education agency) , whether 
evaluation related services (e.g*, staff training) were provided, 
the design of the study (pre- post, mixed, etc.) f and the content 
of the evaluation (e.g. # Chapter 1 program) . v The report 
abstracting form and coding key is located in Appendix B. 

RESULTS AND DISCUSSION 

Evaluation Budget Allocation Patterns 

Prior to comparing the percent allocation figures obtained in 
the two studies, the data had to be transformed due to 4 
differential treatment of overhead costs in the two studies. 
Indirect or overhead rates were assessed as a cost over the fixed 
budget in the SEA study, and as a part of the total budget in the 
contracting firm^ budgets, In order to make the budget category 
percent figures comparable across the two studies, percentages 
for the contracting firm budgets were calculated based rn total 
cost minus the overhead rate. Treatment of overhead cost in this 
way also made the percent figures for our two studies comparable 

7 
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to those in the Alkin-Stecher (1963) scudy. Percent figures for 
each budget category was then based on a total budget which did 
< not include overhead rate. 

Table 1 provides a comparison of the average percent by 
budget category for the two studies. The breakdown for mean a 
percent budget allocations is not particularly surprising. In 
fact, in terms of overall patterns, the data from the two studies 
are fairly consistent. The allocations to personnel and to 
overhead represented the highest costs while all other categories 
received 10 percent or less of the available monies. 

Personnel . Evaluators in both our studies allocated €1 
percent of the budget to personnel. Similarly, in a study of 
evaluations conducted for educational extension Paisley et al. 
(1978) found 63 percent of the budget allocated to personnel. 
Similar allocations have been observed in local budgets (Sartaan, 
Rivenburg, and Moore, 1984). These percentages differ, however, 
from the 86 percent of the budget allocated to staff 
(professional and support) by the Alkin and S teener (1983) 
respondents. 

A closer look at budget categories studied may explain the 
differences of 61 percent versus 86 percent allocations to 
personnel. Indirect cost, whicn increased the total cost of the 
evaluation by 13 percent to 14 percent in our two studies was not 
assessed in the Alk in- S teener (1983) study. It seems plausible 
that the respondents in the Alkin-Stecher {1983} study, who were 
not provided with a category for indirect costs in their budgets , 
may have increased their allocations to personnel to include 
indirect costs- as well* 

Travel. There was a difference between the studies in their 
travel allocations with an overall average of 17 percent of the 
budget devoted to travel by % the contracting firm budgets as 
compared to 7 percent by SEA evaluators. This difference was not 
surprising r however / when it is observed that 80 percent of the 
evaluations conducted by the contracting firm evaluators were 
out-of-state contracts. When instate travel costs were f igured r 
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the travel allocation by the contracting firm was 6 percent* 

Alkin and S teener (1983) respondents allocated an average of only 

3 percent of their budget to travel. 

- #■ 

Table 1 

Mean Percent Budget Allocations 



for Evaluations* 

Budget, Category Study 

SEA Contracting 

Estimates Firm projects 

(n«29) (n«85} 

\. 

Personnel {professional and 

support - salary and benefits) 61% 611 

Staff travel 7% 171 

Consultants 6% -0- 

Materials, supplies, telephone 10%** 6%*** 

Data Processing 6% -0- 

Other 10% 14% 

100% 98%**** 

Overhead added to cost of evaluation 13% 14% 



*The categories of "consultant travel* and "facilities" were 
dropped due to small** percentages • 

**Th is is a composite category obtained by combining the "postage 
and shipping,* "telephone," "printing,", and "duplication* 
categories* 

•••This is a composite category obtained by combining the 
"communication," "duplication," and "supplies," categories. 
★•••Percent does not add up to 100 due to dropped categories. 



Consultants , The contracting firm budgets did not allocate 
any funds to consultants because they themselves were consultants 
and, as a rule, did not subcontract to other consultants. The 
SEA evaluators, however , might be more inclined to contract with 
consultants for evaluation services, thus explaining their 



consultant allocation of 6 percent of the total budget. Alicin 
and Steches's (1983* respondents allocated only 2 percent of 
their budgets to consultants. 

Data analysis . Data processing was not allocated any funding 
by the contracting firm evaluate* s. When queried about this, 
they reported that they usea calculators and secretarial staff 
for data analysis. This approach to data analysis has since bee*i 
changed and their current budgets routinely include allocations 
for data processing. The data processing category was allocated 
6 percent of the budget by the SEA survey respondents a.vJ 
3 percent of by the evaluators in the Alkin-S teener (1983) study. 

Other . The other category was optional for SEA evaluators 
and they used it to "Identify other evaluation costs such as 
equipment purchase or staff training. An average of 10 percent 
of the budget was allocated to this category by SEA evaluators. 
The contracting firm used the other category to cover 
unanticipated costs. On the average* 14 percent of the 1 total 
budget was allocated to this category by the contracting firm. 

Overhead. Although overhead- was treated as a cost in 
addition to the evaluation' budget and therefore was not included 
in the budget percentages discussed above, it is an inevitable 
and important cost that deserves attention. The contracting firm 
routinely added 14 percent to the cost of an evaluation to cover 
overnead costs, Simi? urly, when the SEA evaluators were asked 
what indi ect ^overhead rate would normally be added to the 
cost of an evaluation, they reported 13 percent as an average 
overhead rate. 

This^ summary of evaluation budget averages can be useful as a 

global guide for evaluation budgeting. Alone , however , it 

}* . 

provides little specific direction to the novice estimating a 
budget for the first time or to others needing information about 
factors that may alter "average* budget allocation patterns. The 
next section provides more detail about a number of factors that 
may affect or charge allocation patterns in an evaluation budget. 
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Variations in Evaluation Budget Allocation Patterns 

while th« aaan percent allocations provide an interesting 
picture of typical allocation patterns, a look at the variation 
in the per cane allocations within each category shows that this 
picture is likely to fluctuate. Table 2 shows ranges of budget 
allocations assigned by the SEA evaluators. 

Taole 2 

Range of Budget Allocations for Evaluations 
Assigned by SEA Evaluators' 

Budget Category Range 

Personnel 30% to 88% 

Staff travel 0% to 20% 

Consul cants 0% to 35% 

Materials, supplies, 

telephone 0% to 50% 

Data processing 0% to 20% 

Other C% to 20% 



The point of Table 2 is that there is substantial variation 
that can occur in the estimates provided by the evaluators on 
budget^ category allocations* Alkin and Steelier (1583) found 
similar variations in allocation patterns with ranges of 
allocations of up to 40 percent for professional staff salaries 
Tsii up to 30 percent for all other categories. The next section 

looks at several study characteristics and activities that may 

i 

contribute to the differing budget category estimates provided by 

i 

evaluators* 
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Factors Effecting Allocations to 
Evaluation Budget Categories 



We have looked at average allocation patterns for an 
evaluation study. However , a reliance on average figures as a 
guide for evaluation budgeting nay be misleading, since an 
examination of the variation of "these allocations suggests that 
some evaluation characteristics or activities may affect tne 
proportional allocation to a particular budget category. In this 
section, we identify factors related to budget allocation 
patterns and discuss their effects. We found six such factors, 
which were: 

Fun&Lng level (e.g., $5,000 or $15,000) 
Purpose of the evaluation (e.g., formative or summative) 
Location of the client (e.g., travel distance) r» 
Whether an evaluation related service was provided (e.g., 
training) 

The design of the study (e.g., pre-post or survey) 

The content of the evaluation (e.g., Chapter 1 program) 

Funding Level 

Alkin and S teener (1983) found that funding level had an 
effect on allocations to clerical/secretarial staff salary, 
travel, and data processing. Consequently, we were interested in 
seeing how funding level affected allocation patterns in the 85 
evaluation budgets. Data from the SEA survey on this effect was 
not available, since the evaluation budgets provided in the two 

» 

scenarios were fixed at $10 ,000 * 

In order to look at the effects of funding level on budget 
allocations , the evaluation budgets, which ranged in size from 0 
to $48,499, had to be grouped into funding level categories- 
Three budget categories were forced by grouping budgets according 
to th« following budget sizes: less than $5,000, $5,00Q~$15,QO0, 
and more than $15,000 « Host of the evaluations had budgets of 
less than $5,000 (76 percent? n*65} , with 17.6 percent (n*15) of 
the projects having budgets between $5,000 and $15,000, and only 
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5.9 percent of the evaluations having budgets greater than 
$15,000. These categories were selected based on the 
distribution of available evaluation budgets and to be comparable 
to the $4,000", $10,000 and $2b,000 categories used in the 
Alkin-Stecher (1983) study* 

Personnel. An interesting relationship was noted between 
funding level and allocation to the personnel category. There 
was a distinct difference between the amount of money allocated 
to personnel {professional and support) for low-cost projects 
versus mid-cost and high-cost projects, up to 13 percent more 
money was allocated to personnel for evaluations with budgets 
less than $5,000 than was allocated to evaluations with budgets . 
over $§,000. ' \ 

While Alkin and Stecher {1983) did not? observe, a relationship 

\ 

between funding and allocation to professional staff, they did 
find a linear relationship betwen funding level and allocation to 
cler ical/^ecr etar ial staff with a 5 percent allocation for a 
$4,000 study, a 10 percent allocation for a $10,000 and a 20 
percent allocation for a $25,000 study. 'Eheir respondents 
reported that for low budget projects the professional staff 
conducted fewer activities that required secretarial assistance. 

Such an explanation may also hold for our data, since the 
higher salaries for professional staff may be reflected in the 
^percentage of money allocated to this category for low cost 
projects. For larger projects, more support staff time may be 
involved, but this would not elevate the personnel costs as 
proportionally as would a similar increase in professional staff 
time. 

Travel. Alkin and Stecher (1983} found a linear relationship 

betwen funding level and travel with -no money allocated to travel 

for a $4,000 study, 2 percent to 3 percent allocated to travel 

for a $10,000 study, and 5 percent to 8 percent allocated for a 

« 

$25,000 study. Allocation to travel was also related to funding 
level in the contracting firm evaluation budgets, although in a 
curvilinear, rather than a linear, manner* Recall that average 
travel allocation for evaluations conducted by the contracting 



firm was 17 percent, since 80 percent of their evaluations were 
conducted for out-of-state client*. However, while 13 percent of 
the budget was allocated for travel in a low-cost evaluation, 
this increased to 18 percent for a aid-cost evaluation and then 
back to X3 percent for a high-coat evaluation* 

. while the allocation of lees money to travel for small budget 
evaluations was not surprising, the drop from 18 percent for 
mid-cost evaluations to 13 percent for high-cost evaluations was 
unexpected. One reason for the lower percentage of funds being 
allocated to travel for high-cost evaluations may be the 
proportional relationship of a relatively fixed travel cost to 
the variable budget sizes. That is, a high-cost travel expense 
would represent a high proportion of a mid-sized evaluation 
budget but a smaller proportion of a large evaluation budget* 

Purpose of the Evaluation (formative versus summative) 

Alkin and Stecher expected that " the type of evaluation . . . ' 

will affect the distribution of resources." (p. 26) . 

Consequently, they were surprised to find few differences in 

budget allocations for formative versus summative evaluations. 

were surprised as well, and consequently designed our two 

studies to enable us to better explain the presence or absence of 

a relationship between the purpose of the study and budget 

de- 
allocation patterns* 

Since ^an expected difference in budget allocations is 

predicated upon the assumption that methods involved in the two 

study types differ, we collected data to support or disprove this 

assumption* Analysis of the contracting firm records showed that 

methods used in summative and formative evaluations differed* 

Summative evaluations were more likely to use a pre- post design, 

while a formative evaluation was more likely to use archival data 

and observation. This information, however, was not detailed 

enough to be of much assistance in explaining allocation 

differences for the two study types. 



14 

29 



The SEA respondents were asked to check from a list of IS v 
possible evaluation activities , which would be & part of a 
typical formative study and which would be a part of a typical 
suaaativ* study. Th* results from this checklist are provided in 
Appendix Cm Host evaluation activities war a used in both types 
of s^dies; However, scm specific differences in the methods 
for the two types of evaluation were evidenced* 

A formative atudy was much more likely to employ development 
and piloting of instruments, use of onsite observation and 
interviews, and followup consultations. A summit ive study was 
more likely to include use of 'an evaluation advisory panel and 

expert reviews, control groups, and collection of post test data. 

♦ * 

Once we had determined that: there were method differences, we 
looked at differences in budget allocations for the two studies. 
We found differences in allocations to data processing, travel, 
and personnel according to the purpose of the evaluation. 

Personnel . Intuitively, one would expect that a formative 
evaluation, which requires more active participation by the 
evaluators, might require larger amounts of money devoted to 
personnel costs. This expectation was borne out in the SEA study 
where an average of 5 percent more money was allocated to 
personnel for a formative study. Similarly, in the contracting 
firm budgets , an average off 7 percent more money was allocated to 
personnel for a formative evaluation. - 

A look at the differences between the amount of money 
budgeted for personnel and the final cost for personnel for the 
contracting firm evaluations tells a different story, however . 
The final personnel cost for a formative study was 9 percent less 
than the budgeted cost. In contrast, the final personnel cost 
for a summative study was 1 percent higher than its budgeted 
coat, and 5 percent higher than the final p«?i»Oftjitiffl CO&ic for a 
formative study. In other words, these evaluators overestimated 
the time personnel would spend on a formative evaluation and 
underestimated slightly the time personnel would spend on a 
summative evaluation. In the end, the time spent on a summative 
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evaluation (based on salary paid) exceeded the time spent on a 
formative ^vaJ^uation by 5 par cant. 

A look at the SKA activities checklist associated with a 
formative evaluation provided some explanation for the 
over estimation of personnel coats. While tne formative 
evaluation requires more onsite work, such as observation and the 
conduct of interviews, the summative evaluation requires more 
methodologically complex activities such as use of a control 
group and pre- post designs. The planning and conduct of the 
me thodolog ically complex activities may take more time over th* 
course of the evaluation than the planning and conduct of the 
onsite activities. 

Another difference between the two methods is that a 
formative evaluation is likely to include a followup consultation 
while the summative evaluation provides information to program 
staff and administration in a formal written report. The time it 
takes to write a formal report may explain why a summative 
evaluation requires more staff time* The mean number of pages 
{or a formative evaluation report was 106, with a range of 26 to 
276 pages, compared to a mean number of pages of 111 with a range 
of 0 to 574 pages for a summative report* 

Travel. The amount of money allocated to travel varied 

H 1 1 

according to the purpose of the evaluation. In the SEA study, an 
average 2 percent more money was allocated jz a formative study 
than for a summative study; and in the contracting firm budgets, 
an average of 3 percent more money was allocated to travel for a 
formative study than for a summative study* This trend is in 
keeping with Alkin and Stecher f s (1983) observation that 2 
percent to 5 percent of the budget was allocated to travel for a 
summative evaluation and 8 percent to 10 percent of the 
evaluation was allocated to travel for a formative evaluation. 

Data processing . The SKA evaluators allocated an average 
3 percent more to data processing in summative studies than in 
formative studies. Three percent is noteworthy given that the 
average allocation to data processing is only 6 percent* The 
evaluation activities checklist completed by the SEA evaluators 
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provided some explanation for th^ increase in allocation to data 
processing for summative studies. Kinety-seven percent {n*2S}^ 
of the SEA a valuator* said that statistical analyses would ba 
part of their summative study, while only 79 parcant (n»23> 
included statistics in their formative study* 

Location of Clients 

The atudy of tha contracting firm evaluation budgets provided 

information on tha affect of tha location of clients to budget 

allocations to tha travel category* As mentioned previously, 80 

percent^ of the evaluations conducted by this agency were for 

out-of-state clients. Specifically, the clients of the 

evaluations wera froai nine western states and Micronesia. The 

greatest number of evaluations were conducted for the states of 

Washington (n-3I; 36,5%), Alaska (n»19; 24%), and Oregon (n»17; 

m 

20%) * 

Travel, Data from the 85 cases provided a good deal of clear 
information about the relationship between the location of the 
client and the amount of money allocated to travel. Not 
surprisingly, allocation to travel category was a direct function 
of the travel distance required. Budget allocations ranged from 
6 percent for travel instate (within Oregon state) to 26 percent 
for out-of-state travel. Percentages allocated for travel by 
state were as follows. 

Percentage 



Allocated State (s) 

6% Oregon 

12% Washington 

19-21% Alaska, New Mexico, Arizona, Micronesia 

24-26% California, Idaho, Montana, Wyoming 



The percentage of money allocated to travel was highest in 
rural states of Wyoming, Montana, and Idaho, where use of 
expensive commuter planes is usually necessary. 

Total budget . The location "of the client also affected the 
total budget of the evaluation for mid-range evaluations. 
Evaluations conducted in nearby or adjacent states had lower 

overall budgets than did evaluations in more distant statetf* 
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Provision o£ gvalu.ation~ReXat.e3 
Services 

For eacn report associated with a contracting firm evaluation 
budget, it was noted whether an evaluation-related service was 
provided. Such a service typically consisted of assistance in 
program goal definition, assistance in program implementation r 
training for staff in evaluation practices* etc* About half of 
the reports (n»41? '48.2%) documented the provision of 
evaluation-related services in addition to the conduct of the 
evaluation. We found that evaluation-related services were . 
related to th# budget in several w^ys. 

Personnel. One would expect that the provision of 

t 

evaluation-related services would increase the amount of money 
allocated to personnel. In fact, the opposite was found. Five 
percent more money was budgeted for personnel if no evaluation- 
related services were provided. This trend may be related to the 
type at study. A formative study would .involve more evaluation- 
related services by definition because of its interactive and 
exploratory nature* As noted earlier, however, in the 
contracting firm evaluations, le&s money was budgeted and spent 
on personnel for a formative study than for a summative study. 
If evaluation related services are associated with formative 
studies, then the allocation^ of less money to personnel when 
evaluation related services are not provided is consistent with 
the expected pattern. 

Travel , The provision of an evaluation-related service must 
entail travel, since an average 6 percent more money was budgeted 
by the contracting firm-on travel if evaluation research services 
were provided. v 

Overall budgets While a higher percentage of money was 
budgeted and spent on personnel if evaluation-related services 
were not provided, the total cost of a study that involves 
evaluation-related services is higher than a study with no 
additional services. The provision of evaluation-related 
services elevated the total budget by an average of $2 ,000 . 
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Within each of the major evaluation projects, cue or two 
actual evaluation studies Might appear* This resulted in a total 
of 118 evaluation studies. Usually a report that described more 
than one study began with a formative evaluation and concluded 
with a summative evaluation. 

Each evaluation design was categorized into one^of seven 
groups; pre- post? archival data? mixed (pre- post, interviews, 
surveys , and/or arcnival data); pre- post and pilot study; 
development of an observation guide; pre or post only, or 
survey. Of these Methods, mixed (n-35; 30%) , ^archives (n*30; 
25%) and pre- post (n**27; 23%) represented 78 percent of the 
designs. 

Personnel, < h pre- post design affected personnel funding 
allocation with between 6 percent and 11 percent sure money 

7 

budgeted to personnel for this design than for any other design. 
Actual expenditures for personnel in a pre-post design was 
between 2 percent and 3 percent higher than actual personnel 
costs for any other design. Clearly, a pre-post evaluation * 
design required more time from the aval ua tor . ~ * 

The conduct of an evaluation which developed an observation 
guide for program implementation was also associated with higher 
personnel costs. When an observation guide wajs developed , 
6 percent c&cre money was spent than was budgeted for personnel. 
The reason for underestimating personnel costs when writing an 
observation guide may be the same as the reason tor 
underestimating personnel costs in a summative evaluation — 
unanticipated ataff time spent writing. 

Looking at the number of pages associated with the 
observation guide reports supports the writing-time hypothesis. 
An observation guide had an average of 145 pages compared to a 
pre-post design report which averaged 105 pages, an archival 
study report which averaged 108 pages, and a mixed design report 
which averaged only 99 pages. 
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Travel- Analyaifc of archival data arid a*lx<sd design 
evaluations were associated with higher travel costs* Presumably, 
because these assigns would require ffore on-site visits, an 
average of S percent sore aoney was 'budgeted and 4 percent store 
money was actually spent- on travel costs. 

Total cost . A pee-post design also affected the total* cost 
of the evaluation, An evaluation with a pre- post analysis was 
oudyeked xor an average of about $2,000 more, and actually cost 
about $2,165 more, than did any other analysis. Thirty- three 
percent of the pre-post studies were budgeted for $5,000 oc more, 
while only about 20 percent of tfce other studies were budgeted at 
$5,000 or more. 

r l 

Content of the Program ' 

The relationship of the topics or content of the programs 
being evaluated with budget allocation patterns » \s also 
explored. Of the 85 evaluation reports,- 10 content categories 
were identified. Of these content. areas, Title I {Compensatory 
- Education, now known as Chapter 1) , Title IVA (Indian Education) , 
and Title IVC (Improvement in Local Practice) represented a the 
majority of content areas {n*52; 60%). « When compared, the 
allocation patterns for these three content areas were nearly 

r 

identical. However, the total cost of the evaluations for these 
programs differed greatly. 

Total cost . The total cost of the three main content areas 
were quite different. The mean cost of an evaluation of a 
Title I program (n*I3) was $12,104, the mean coat of an 
evaluation of a Title IVA program (n*2S) was $2,684, 4 and the mean 
cost of an evaluation of a Title IVC program (n-13) was $4,661. 
One reason for this difference in total cost may oe that Title I 
evaluations covered a longer time span or included both a 
formative and a summative evaluation. 
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CONCLUSION 

I 

Although budgeting is a critical part of every evaluation, 
thtre is very little empirical data to guide the practice of 
estimating the budget for an evaluation. Tne Ailcin-Stecher 
(1383) study* one of the first attempts to identify typical 
allocation patterns within evaluation budgets, provided a natural 
starting point from which to launch our studies of evaluation 
budgetings By coaparing our study result* with the Alkin and 
Steelier (1983) results, we hoped to find zom commu* allocation 
pattern* for evaluations, to provide gui< e lines for establishing 
accurate evaluation budgets , and to identify oitfalls particular 
to evaluation budgeting. In fact, a general evaluation budget 
pattern was observed and conation factors affecting allocation 
patterns vera identified. 

Me recognize that it is difficult, however, to know whether 
to attribute these patterns to true budget characteristics or to 
me thodoiog ical artifacts. Certainly much has been written on the 
problem of combining data frost independent studies- in order to 
attain stronger conclusions (e.g., Slav in, 1984} Cook and 
Levi ton, 1980; Pillener and Light, 1980). This review, however, 
has not combined data, but rathe" compared it*, Consequently, 
that similar patterns and effects have emerged despite 
methodological and population differences supports the notion of 
a common budgeting pattern, which say provide a useful starting 
point in a variety of evaluation contexts, 

in terms of generalizations that could be useful in the 
practice of evaluation budgeting, the SEA estimates of budget 
allocations and the contracting firm evaluation budget figures 
both suggest that 61 percent of the budget might be a good 
guideline for estimating personnel costs. In order to accurately 
estimate per sonnel^ costs , of course, several factors should be 
considered before assigning a final budget figure. For example, 
total size of the budget should be considered. In a small 
evaluation, a higher proportion of the evaluation costs are going 
to be in the form of personnel expenses. Also, if the evaluation 

21 

27 



employs a pr*i- post design or If considerable writing is required, 
up to 3 percent to 11 percent sore money may need to be allocated 
to personnel* Even with this higher allocation, the contracting 
firm budget* often underestimated actual personnel costs by 
6 percent. 

These data also suggest that it is common to overestimate 
personnel costs for a for native evaluation. Both the SEA 
estimates and the contracting firm budgets allocated more money 
to personnel for # formative evaluation. However, the actual 
personnel costs for formative evaluations, according to the final 
cost of data for the contracting firm evaluations, was much less 
than anticipated* This over estimation of personnel needs, albeit 
not as serious a& an underestimation, does not contribute to 
optimal use of evaluation funds. 

While allocation to travel was also affected by a number of 
factors, it was accurately anticipated in the budgets and 
therefore did not constitute a pitfall for the evaluators. Of 
the factors that did affect the travel budget, the most obvious 
was the effect of the total budget sise on travel allocations. A 
sma ll evaluation budget simply could not accommodate the luxury 
of many on-site visits, particularly if out-of-state travel is 
required. Those factors related to increased allocation to 
travel were the conduct of a formative evaluation, the provision 
of additional evaluation-related services, and the use of 
archival data or mixed designs. 

Allocating accurate amounts of the budget to data processing 
also did not present a difficulty to evaluators in that the 
amount budgeted was comparable to the amount spent at the end of 
the evaluation. Allocation to data processing, like travel, was 
somewhat related to the total budget size. If the evaluation 
project was small, then less money was available for data 
processing. If a summative evaluation was conducted, more money 
was allocated to data processing. 
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Finally, the total coat of an evaluation was related to a 
variety of study characteristics. Travel distance, the 
complexity of the evaluation design, the provision of evaluation- 
related services, and the content of the program being evaluated 
were ail related to the total cost. 

The results of this study are not earth-shattering. Rather, 
they provide a a tart on a useful guide for the development of 
evaluation budgets in a time when dwindling resources make budget 
accuracy very important. The guidelines provided in this paper 
should be considered as tentative, however, until more evidence 
can be collected. Future studies should attend to the many 
sources of budget data currently available in order to make 
comparisons between projected budget estimates and final 
evaluation expenditures, and to identify factors related to 
budgeting patterns. It is with these types of comparisons that 
we can discover evaluation budgeting pitfalls and remediate 
inaccurate budgeting procedures in evaluation. 
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APPENDIX B 

« 

Stpoct Ab*tr*cti&g Tosm 



V3UtXASLS KJU4K - . coos COLEMW 


* 

Titles 




• 

1-2 




Beport Codas 


Principal investigators 




3-4 


Client! 






contents 




7-8 


Prograa developments Yes * I No m 2 




9 


Evaluation related services* Yes » 1 Ko ■ 2 




10 


Conduct of evaluations Tea » 1 Ho ■ 2 




11 


Focus It Process * 1 Outcome « 2 




12 


Purpose Is Formative » I Suao&tive « 2 




13 


Designs 




14-15 






16-17 




Muaber of instruments developed* 


Total nunber developed instrument* that vara administered 




18-20 


number standardised 1 as trumsnts uaadis 




21-22 


Total nuxabar standardised inetruments that vara administered 
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Pecos 2s. Process - X Outcoaa » 2 


• 
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Purpose 2s Tentative « 1 Susoativs • 2 
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Designs 




23-29 






30-31 




Muaber of instruaents davslooads 


Total number developed instruaents that were administered 




32-34 






3S-3S 


Total number standardised instruments that were admiaiaterad 




37-39 


Otner Baaearcfes 




40-41 






42 


Muaber reports delivered 


Total Muaber of r sport pages* • 
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XPFEKDIX C 



Percentage of S&A ©valuators iaiicating the element that would be 
a major part of a fcaaative or a suamative evaluation 

Formative Suaaative Evaluation Element 

(n) (a) \ 



$6.6* 
(28) 


69.O* 
(20) 


Development of an evaluation plan 


27.656 
(8) 


44.8* 
(13) 


Use of an evaluation advisory panel 


79.3* 
(?3) 


37.9* 
(U) 


Development and piloting of instruments 

r 


27.656 
(8) 


(17) 


Use of control or comparison groups 


79. 3* 
(23) • 


65.5* 
(19) 


Collection of pre-test data 


75.9* 
(22) , 


51.?* 
(15) 


Collection of on-site observation data 


75.956 
(22) 


65.5* 
(19) 


Review and analysis of documents 


'9.<# 
^20) 


51.7* 
(15) 


Collection of personal interview data 


17.256 
(5) 


31.0* 

(9). 


Use of phone interviews 


2?.6* 

(8) 


34^5 

(iofv 


Collection of mail survey data 


24.1* 

(7) 


20.7* 
(6) 


1 

Use of community hearings 


13.8* 

w 


46.36 
(14) 


Use of expert reviews 


62.1* 
(18) 


93.1* 
(27) 


Collection of post-test data 


24.1* 
(7) 


24.1* 
(7) 


Eevelopment of case study reports 


79.3* 
(23) 


96.6* 
(28) 


Statistical analysis of data 
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Appendix C continued 



Formative 

% 

(n) 



Sufttt&tive 
% 

M 



El eaent 



(21) 

(27) 
(22) 



65.5^ 

(19) 

93*1$ 
(27) 

55.2* 
(16) 



Verbal presentation of study findings 



Developeausmt of narrative reports 



Provision, of consultation and £ oliowup assistance 
to project staff 
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